Repository navigation
Epic: zarr serving next phase - #1206
Merged
Merged
Conversation
* Phase 1: edge cache for zarr data plane Range-keyed cache entries (single bytes-range, synthetic 200 storage for the 206 the Cache API refuses), a canonical origin+path+v cache key, tokened Cache-Control (zarr.json/index.json get a day when ?v= is present, index.json's untokened TTL rises 60s -> 3600s), edge-cached missing-object 404s, and a deps seam (cache/fetch/s3Base) so backend/test/zarr-data-cache.test.ts can drive the real route against a real Bun.serve() upstream and an in-memory cache. Addresses #1178 (items 3, 4) and #1035. Part of epic #1181. * Address PR #1199 five-reviewer follow-up Correctness: gate isPublicDataset() before every cache lookup, not only on miss, so a dataset flipped private stops serving immediately instead of up to a day later; a ranged 404 lookup now falls back to the canonical negative-cache entry so a ranged probe storm converges; a 206 with no Content-Range fails closed (stream through, warn, don't guess); cached entries get a size cap (8 MiB range / 32 MiB full), checked from Content-Length before any clone/put; 416 now passes through as 416 with Content-Range instead of collapsing into 502; cache.put failures are caught and logged instead of risking an unhandled rejection under waitUntil; app.onError logs and still applies CORS instead of a bare 500. Cache key and TTLs: canonicalCacheUrl now derives the key from the normalised S3 key (each segment encodeURIComponent'd) instead of the raw request pathname, so differently percent-encoded spellings of the same object collapse onto one entry; untokened index.json drops to 300s/3600s (was 3600s/86400s) with a corrected comment on why (the zone id is unset in prod today, so the purge is a no-op and the TTL is the only bound); the chunk-purge comment now says zarrPurgeTargets never targets a chunk URL. Types: NormalizedRange/CanonicalCacheUrl are branded strings; cacheControlFor takes an options object; the test InMemoryCache throws on a 206 (mirroring the real Cache API) with a comment on why only 206. Tests: 45 (was 29) in zarr-data-cache.test.ts, covering every item above plus a v-token round trip, telemetry via a real binding-shaped collector, four concurrent identical range requests, HEAD with a Range header, and an upstream that ignores a well-formed single range. Part of epic #1181 Addresses #1178 (items 3, 4) and #1035 * test: update cacheControlFor route test to the new contract * fix: absorb synchronous cache.put throws
* Add --test and --print-config to hallu-zarr.sh * Add config-resolution tests for hallu-zarr.sh --test * Run hallu-zarr.sh config tests in CI * Thread TEST_API_URL through --test for nemar CLI * Document the Zarr test conversion instance * Add validated recipe for hallu-zarr.sh --test * Point AGENTS.md Zarr paragraph at test instance * Harden --test guard rails against review findings * Add tests for hardened --test guard rails * Fix guard-rail count in CI comment * Amend ADR 0029 on hallu-zarr.sh self-deploy * Fix Zarr test-instance pointer wording * Correct 3.3/3.4 ground truth and wording * Add bootstrap-copy gotcha; reflow section 9 prose * fix: strip DNS root dot in prod host guard * fix: require bash 4+ for the host guards
* Add has_zarr dataset filter * Expose zarr fields in catalog responses * Add zarr-catalog document service * Add admin route to publish zarr catalog * Publish zarr catalog on daily cron * Serve zarr-catalog.json on zarr host * Add CLI has-zarr flag and publish command * Test has_zarr filter and catalog zarr fields * Test the zarr-catalog document service * Add contract tests for zarr fields * Add CLI test for has-zarr flag * Log malformed zarr_data_failures input * Defend zarr-catalog service invariants * Remove test-only endpointUrl from S3 type * Require zarr_index_url only when ready * Name the zarr-catalog dev-cron allowlist * Add publish DI seam to admin zarr route * Clarify has_zarr filter's stable meaning * Test hardened zarr catalog fields * Test zarr-catalog service invariants * Test the zarr-catalog publish route * Pin publishZarrCatalog cron wiring * Test the zarr_index_url invariant * Let fetchZarrCatalogObject take a fetch impl * Wrap serveCatalog signature (biome) * Test GET /catalog.json via the phase 1 harness
* feat: redirect zarr bytes for non-browser clients Non-browser GET requests to a zarr store object (no allowlisted Origin) now 302 to the public S3 object instead of proxying through the Worker, never touching D1, the edge cache, or an upstream fetch. index.json stays proxied/edge-cached/D1-gated, HEAD is never redirected, and the rate limiter exempts redirect candidates from the data-ip bucket via the same predicate serve() uses to route. Closes #1061. Part of epic #1181. * fix: encode S3 key segments in s3PublicUrl s3PublicUrl interpolated the decoded key unescaped, so a literal '#' truncated the redirect Location at the fragment, '?'/space broke the request line, and a non-Latin-1 character made workerd refuse to send the header at all (Bun tolerated it, so tests stayed green). It now shares canonicalCacheUrl's per-segment encodeURIComponent treatment via a new encodeS3KeyPath() helper, so the fetch target, the cache key, and the Location all agree. Exported s3PublicUrl for a direct unit test of its real (non-s3Base) branch. PR #1202 review item 1. * fix: anchor redirect path regex and validate dataset id ZARR_OBJECT_PATH_RE's \/zarr\/?(.*)$ matched "bogus" as rest for /on000001/zarrbogus, exempting a path that never reaches serve()'s route from the rate limiter. Anchored on \/zarr(?:\/(.*))? like ZARR_PATH_RE, requiring end-of-string or a literal slash after zarr. isRedirectCandidate never validated the dataset id either, so zz000001 (wrong prefix), nm1234567 (too many digits), and nm999999 (over the 99999 cap) got redirected to a garbage S3 URL and wrote an invalid id into Analytics Engine, while the same request with a browser Origin correctly 404'd. The regex now captures the id and isValidDatasetId gates it; a malformed id falls through to the normal proxied 404 path instead of a separate D1-free branch. Also drops the stale "catalog.json will need the same exemption" comment: catalog.json is a separate route with no <id>/zarr/ segment and can never match this regex at all. PR #1202 review items 2, 3, 7 (comment only; item 7's tests follow). * fix: bytesFromRangeHeader never returns negative bytes "bytes=100-50" is syntactically a valid single range (parseCacheableRange only checks digit shape, not end >= start), so it computed -49 and that landed in the Analytics Engine data point. Real S3 would 416 an inverted range like any other out-of-range request, so there is never a real byte count for one; fall back to the same "unknown" 0 as every other unmeasurable case. PR #1202 review item 4. * feat: observe-only rate-limit counting for zarr redirects Replaces the unconditional per-request console.info the zarr redirect exemption used to emit with an observe-only mode on rateLimiter itself (a new RateLimiterOptions.observeOnly). A redirect candidate now counts toward the SAME IP-keyed data-ip bucket a proxied request from that client would be enforced against, but never blocks -- so a later proxied request from a client that's been hammering the bucket via redirects still 429s correctly. Logs at most one console.warn per window, the first time the bucket crosses the point enforcement would have tripped, with a hash of the IP (not the raw address) rather than a log line per exempted request. Updates AGENTS.md's zarr paragraph to match, and corrects a related inaccuracy: catalog.json is proxied and edge-cached but has no D1 gate of its own (it lists only already-public datasets), unlike index.json. PR #1202 review item 5. * docs: document chunk-redirect in access-metrics detail contract AccessEvent.detail's zarr contract listed only index/metadata/chunk; the redirect branch's "chunk-redirect" marker (#1181 phase 6) was undocumented at the source of truth for the field. PR #1202 review item 6. * test: catalog.json is proxied, not a redirect candidate Route-level companion to the isRedirectCandidate unit tests added earlier: a bare GET /catalog.json still gets a 200 from S3 via serveCatalog, never a 302 -- this route has no <id>/zarr/ segment and never consults isRedirectCandidate at all. PR #1202 review item 7 (tests). * test: cover redirect edge cases from review - /zarrproxy + allowed Origin + store object: proxied bytes with CORS (item 8) - HEAD tests assert the fake upstream actually received method HEAD, not just a matching path/range (item 9) - Private dataset redirect safety: follow the 302 Location for real and confirm S3 itself 403s an unauthenticated request under the private prefix (item 10) - 8 concurrent redirect-candidate GETs: all 302, zero upstream, zero D1 (item 13) - Re-verify range-key hit/miss, the full-object size cap, and the 206-without-Content-Range fail-closed path via a no-Origin index.json request, since those mechanics moved to allowed-Origin store objects earlier in this PR (item 12) Items 1-7 and 11 landed in earlier commits on this branch. PR #1202 review items 8, 9, 10, 12, 13. * refactor: hash observe-only IPs with hashIp
* feat: zarr fidelity verification sweep service * feat: admin route for zarr fidelity sweep * feat: wire fidelity sweep into daily cron * feat: add has_zarr_verified filter * feat: add zarr verify fields to contract * feat: expose verify fields in zarr catalog * feat: add zarr-fidelity-sweep CLI command * feat: add --has-zarr-verified CLI flag * docs: document fidelity sweep in AGENTS.md * fix: harden zarr fidelity sweep per PR 1203 review * fix: 502 when every sweep candidate errors * feat: surface new sweep fields in the CLI * test: has_zarr_verified filters GET /datasets/search * docs: update fidelity sweep paragraph for review fixes * fix: scope budget_exhausted to the sweep-wide budget
* Pin biosigio 1.2.6 and add jsonschema * Add zarr index v3 and manifest schemas * Publish zarr index v3 with coverage data * Requeue done datasets with pending retries * Pass contract base and api base to the driver * Count pending zarr recordings in the backend * Serve the zarr index schema from the API * Test index v3 consumers and the schema route * Document index v3 and the engine-bump steps * Pass the sidecar to both zarr exporters * chore: bump zarr engine to 3 * ci: install jsonschema for the zarr python tests * Type the index entries and build failures once * Keep legacy non-raw stores in the index * Redact credentials from published failure detail * Drop non-raw stores loudly and report the count * Test that refused publishes still call back * Spare untried recordings from the retry budget * Close the schemas and publish the layout recipe * Validate the zarr callback and cross-check coverage * Cover the clean-run, entry assembly and guard paths * Document the MCP layout recipe * Reconcile the non-raw purge wording * test: pin the CLI default limit; redaction false positives
* Publish events.parquet with sample indices One columnar file per dataset at <id>/zarr/events.parquet, one row per (event, channel group), with the level-0 sample index computed by the converter -- the only party that knows the exact resampling relation (#1060). index.json gains events_parquet + events_row_count and a layout entry; n_events/trial_types now come from the same parse as the rows. Rows are staged per store on disk as recordings finish and written store by store into a ParquetWriter, so a 25k-store dataset never builds one table. An incremental run carries an untouched store's rows forward from the published file exactly as the index carries its entry. Best-effort like manifest.json: a missing pyarrow or a failed build/upload warns, leaves the index naming no file, and reports events_upload_failed on the callback rather than failing a good conversion. Tested: converter suite (438), incl. a main()-level clean + incremental run over a real EDF, and mutation-checked on the formula, the ordering, the flush, the carry-forward and the schema. * Verify sample indices on real recordings nm000329 (1000 Hz served at 250 Hz) is #1060's acceptance case, and a fixture built at the serving rate cannot check it: the error class the column removes only exists when the rates differ. Downloads the recording and its events.tsv on demand into a cache outside the repo (skipped, never failed, when offline; NEMAR_ZARR_SKIP_REAL_DATA opts out), converts it with the real exporter, and checks three things that do not reuse the formula: level-0 n_samples == round(n_native * target/native), zero filter delay (served level 0 correlates with native[::4] peaking at lag 0), and every onset within one sample of the level-0 grid -- exact except on 13 events that sit exactly half a sample between two, where no rule can do better. The dataset's own native `sample` column agrees on the same terms. Adds a synthetic 512 -> 250 case (ratio 125/256), since nm000329's exact 4 cannot tell a formula that assumes integer decimation from one that does not. * Add pyarrow to the converter dependencies The events.parquet writer needs it (#1060), and the schema/ordering/ bounded-write tests are pure Python, so the fast zarr-python-test tier installs it too -- otherwise those tests do not fail, they skip, and the converter's "warn and publish no events file" degradation makes a broken writer look like a dataset with no events. hallu-zarr.sh's fallback spec is unchanged: the venv installs from requirements.txt. * Record the events file in the producer manifest manifest.json now lists the events.parquet a run uploaded with its byte size and row count, so "the object on S3 is the one this conversion wrote" is a HEAD away rather than a download. Not carried over from the prior manifest: a run that published no file must not claim the previous run's bytes. * Type the events fields on the zarr callback events_row_count (nullable) and events_upload_failed were reaching the backend through passthrough(), so nothing said what they mean or that null is a real value. Declared, documented, and tested; deliberately not persisted -- the row count is served by the published index, and a column would spend the datasets budget (ADR 0034) on a number already available. * Cache and purge the zarr documents as one set manifest.json was getting the 24-hour chunk TTL and no purge, so a re-conversion's manifest could disagree with the fresh index at the edge for a day; events.parquet would have inherited the same gap. One ZARR_DATASET_DOCUMENTS list now drives both cacheControlFor's short untokened TTL and zarrPurgeTargets, so the two cannot cover different files. Only index.json stays always-proxied: the other two redirect to S3 for non-browser clients like any store object. * Say why the telemetry vocabulary is narrower zarrObjectType's comment points at cacheControlFor, which now covers three documents while this still labels only index.json -- on purpose, since "index.json is 95% of egress" is a measurement of index.json. * Update the purge-target tests for the document set The canonical zarrPurgeTargets tests live in test/, not backend/test/, and asserted the old index.json-only list. They now assert all three documents against the shared constant; the backend cache file keeps only the cross-module claim (every purged document has the short TTL, no chunk does), which is the half that belongs beside cacheControlFor. * Drop the events pointer when a purge rewrites the index The published events.parquet still holds rows for the stores the purge just removed, so events_row_count would overstate it and events_parquet would name a file describing stores the index no longer lists. Both go until the next conversion republishes them, which is the same rule the converter follows. * Fix the events step's carry-forward and ordering Three defects in one code path (they do not build apart), all from review of #1205: Carry-forward was defined as "produced no rows this run", so a store that WAS reconverted and legitimately has no events -- its events.tsv deleted or emptied -- was treated as untouched and had its stale rows republished from the prior file forever. On a first --clean run the same definition reported every event-less store as an unrecoverable carry-over. `reconverted` (every store this run rebuilt, rows or not) is now threaded from `record` through to the writer, which never falls back to prior rows for one. Ordering: the index is now validated BEFORE anything is uploaded, in the shape it will take (the events pair filled with the deterministic URL and a `minimum: 0` placeholder count), so a refused index no longer leaves a freshly overwritten events.parquet that the live index does not describe. The one interleaving that can still overwrite -- a document that passes with the placeholder and fails on the real count -- now reports events_row_count/events_upload_failed on the failed callback, which is the only place an operator would learn the object was replaced. Per-store signals: a store whose events cannot be positioned is named (`::warning::`) and counted into a new `events_stores_without_rows` callback field -- either every sample_index is null, or the store has events and no channel group to attach them to. The decision is a pure helper because the second case cannot be reached through a real conversion. Tested: 470 converter tests, incl. a mixed dataset through main() twice, the refused-index interleavings, an unclamped past-the-end onset, a failed events upload, duplicate onsets, and a run with pyarrow absent from sys.modules. Mutation-checked per fix. * Log the sibling documents on the zarr callback The summary line now carries events_row_count, events_upload_failed and manifest_upload_failed, and an unpublished sibling gets its own warn. Neither flag has a column and neither fails the run, so without this the condition -- index.json published while manifest.json or events.parquet was not -- was visible only in a cron log on one box. * Write the purged index conditionally purge_non_raw_stores read index.json, spent minutes deleting objects, then PUT the rewrite unconditionally -- so a converter run that published in that window was silently reverted, taking every store it had just added with it. The live document is now re-read (with its ETag) immediately before the write, the rewrite is recomputed from it, and the PUT carries --if-match; a 412 is reported as index_rewrite_conflict for that dataset and the newer document is left alone. An absent index writes with --if-none-match, so a first index published meanwhile is safe too. Both modes now share one planner (plan_live_index_rewrite, renamed from plan_snapshot_index_rewrite): the snapshot path was already doing this, the live path had the same exposure on a shorter fuse. Tested end to end against a real stub aws with md5 ETag semantics: an uncontested rewrite, a concurrent publish detected and not clobbered, a store published between the reads surviving, and the absent-index case. * Log events_stores_without_rows on the zarr-ready callback
This was referenced Sep 2, 2026
* Validate zarr-ready counts as integers Every number on the zarr-ready callback body is a count, so the schema primitive carries .int(): a fractional store_count validated cleanly and was bound into an INTEGER column as a REAL. .catch(undefined) makes it behave like any other malformed field -- dropped and logged, the rest of the body still persisted. Also retypes data_failures as unknown[] to match its validator (nothing reads an element's fields, only the length), and puts not_attempted_count and non_raw_dropped on the summary log line: both were validated and then read by nothing, and neither has a column. * Note the zarr index trust boundary and one cast The index.json shapes in s3.ts and zarr-fidelity-sweep.ts are all-unknown because the document is fetched from the serving bucket, written by a cron on another host: the coverage equation is the producer's invariant (check_index_invariant), not something these types assert. Says so, and says where each file re-derives rather than trusts it. Also tags zarr-data.ts's rateLimiter context cast with SAFETY: it widens the static Variables map on the same runtime object, and rateLimiter reads no Variable at all. * Correct the dev-D1 cron allowlist rationale The comment called dev D1 a partial production mirror; the catalog was purged to the curated fixtures. The allowlist survives that purge for the reasons AGENTS.md gives: the users table was not purged, the dev worker holds a real RESEND_API_KEY, and the nemarDatasets org is shared. * Publish index.json as a conditional write Two processes write <id>/zarr/index.json -- a converter run and purge_non_raw_stores.py -- and both read it, work for minutes to hours, then write back what they merged from that read. The converter's PUT was unconditional, so whichever finished second silently reverted the other, both exiting 0. The PUT now carries --if-match with the ETag read at the top of the run (--if-none-match '*' when there was no document). A 412 is re-read, re-merged onto the newer document and retried once; a second 412 abandons the publish through write_failed_callback so the driver posts a failed callback rather than leaving D1 at converting forever. An upload failure now takes that same path instead of escaping as an uncaught RuntimeError. read_index_with_etag/write_index move into generate_zarr.py and the purge imports them: the two writers of one object must agree on how the ETag is read and sent back. The ETag now comes from the PUT's own response rather than a follow-up head-object, which could observe another writer's version. The stub aws in test_generate_zarr.py gains the real ETag semantics the purge test stub already had. Same file, same pass: PendingEntry.reason becomes the schema's closed three-value enum (normalized through _pending_reason when read off a published index, so a bad value cannot fail validation at the END of a conversion), EventsPublication.file becomes a ManifestFileEntry mirroring the manifest schema, and a duplicate import is dropped. * Reconcile the staging cron and two ops claims systems-inventory and validated_workflows contradicted each other on the staging nightly cron; both now carry the installed crontab line verbatim, including the explicit ZARR_DRIVER_REF override and the note that it flips to dev once the epic merges. Also corrects the --test guard-rail claim (only API_BASE and TEST_API_URL are case-folded; the dirs are path-normalized and the bucket/profile are exact equality) and hallu-zarr.sh's 'no writes' claim for --preview-engine-bump, which connects and so runs migrate_schema. * Serve the zarr summary keys D1 already stores The callback wrote pending/discovered into zarr_data_failures (#1197), the catalog projection kept only count/detail_ref, and the contract stripped what it did not declare -- so counts that existed in D1 reached nobody and every layer's own tests passed. Projects them, declares them, and adds events_upload_failed and manifest_upload_failed to the same bounded object: an index published without its sibling was previously visible only as a console.warn in the Worker log. Both are written only when true, so a later run that publishes both clears the claim. No new columns (ADR 0034). Tested end to end -- real callback route into a real D1, out through the real detail route, parsed by the shipped contract -- which is the only shape that fails when one of the three layers is missed. * Stop erroring datasets starving the fidelity sweep Candidacy clears only on a verdict, and the batch was ordered by dataset_id, so ~25 alphabetically early datasets that error every run (unreachable index, rotated credentials, their own fetch budget spent) held the whole 25-row quota forever and nothing behind them was ever swept -- while each run reported errors and looked like progress. Every attempt now stamps zarr_verify_attempted_at, verdict or not, and the batch is ordered by it (never-attempted first, then oldest attempt). An unreachable dataset still records NO verdict and stays a candidate; it just costs one slot per cycle instead of all of them. The sweep_stamps JSON paths move into one module the sweep and its three readers import: json_extract returns NULL for a path that matches nothing, so a typo read as 'never swept' forever with no error anywhere. A seam test now runs the real sweep and then both readers' real SQL. * Fail loudly when an S3 listing fails _s3_child_prefixes returned [] for both 'no subdirectories' and 'the call failed'. _rm_recursive is correct either way (it falls back to one unsharded delete, and keeps that tolerance at the call site), but list_dataset_ids read the same [] as 'this bucket holds no datasets' and discover_excluded_stores as 'no stranded stores here' -- a clean empty result for a run that could not see the bucket. It now raises, and purge --all with zero datasets exits 2 like the snapshot-directory check: the bucket holds ~800 prefixes, so an empty listing is a misconfiguration, not an all-clean verdict. Also adds the missing purge test for a non-412 write failure (classified as an error, never as a conflict and never as a rewrite), and fixes the editor diagnostics on the converter: typed containers where the values really are TypedDicts, a str head_commit, and two unused bindings. * Say what a 403 and a redirect row actually mean The fidelity sweep reported an unfetchable index as 'index.json is absent'. S3 also answers 403 for a missing key when the caller cannot list the bucket, so a rotated credential or a changed bucket policy produces that same message for every candidate at once; the message now names both readings so an incident is not read as a per-dataset converter bug. zarr-data.ts's chunk-redirect access rows carry the byte count the Range header ASKED for, recorded before the 302, so they are intent and not delivery -- said so at the call site. shared/facets.ts now lists hasZarrVerified among the bespoke filters it excludes. Pins the redirect-before-cache ordering: a chunk already primed by an allowlisted Origin must still 302 for a library request. Nothing else in that file fails if the order inverts. * Compile the published zarr schemas in CI shared/zarr-index.schema.json and zarr-manifest.schema.json are what the converter validates every document against before upload, and nothing on the TypeScript side ever compiled them: a broken schema DOCUMENT (a typo'd keyword, a dangling $ref) failed at the next Hallu run with every TS test green. Adds an ajv test that compiles both, validates checked-in v3 fixtures, and proves the schemas discriminate by mutating one field each. The consumer fixture in backend/test/zarr-index-v3.test.ts now reads the same file: the inline one carried v1's source_key and neither layout nor updated_utc, so every consumer assertion about 'a v3 index' was made against a document the producer would have refused to publish. * Run the converter suite CI actually skips zarr-python-test installs only jsonschema and pyarrow, so five classes -- 43 tests covering real-recording conversion, events.parquet publication, convert_one and the sample-index rate cases -- raised SkipTest on every CI run and were reported as 'skipped=9' in a green job. patch-duration-test already installs the real stack, so the full suite runs there now, under pytest, with a tripwire that fails the job above two skips (the real-data download and the bash-3.2 test). Adds the missing shared/** paths filter: a PR touching only shared/** -- the wire contract, the facet vocabulary, the two JSON Schemas -- matched no filter and ran zero jobs. A tripwire test now asserts every filter is exported as an output and that shared gates the jobs that compile it. Also covers the engine-bump ack file in bash: armed for exactly one run, consumed before reconcile, and the pending-ack notice re-raised as its own error line. * State the real sample_index formula Both the docstring and AGENTS.md said round(onset_s * rate); the code is math.floor(onset_s * rate + 0.5). They differ at an exact .5 tie, which is not exotic (any onset landing on a half sample at the serving rate hits it): Python's round() is banker's rounding and ties to even, this ties up. A client reimplementing the column from the docs would disagree with the published value on exactly those rows, which is the failure the column exists to prevent. Written out verbatim in both places. * Log provenance_fetch_failed, fix four comments provenance_fetch_failed was validated and read by nothing, like the two counts the earlier pass added: it says this wave's stores carry null doi/license/citation/hed_version because the CATALOG READ failed, not because the data lacks them, and it has no column. On the summary log line now, with the test extended. Comment corrections: the garbled 'on008083 shape' becomes 'pending-with-no-failures shape'; application/schema+json is registered by RFC 9358, published, not a draft; the purge script's 'PR #98' is qualified with the repo it was filed in (nemarDatasets/.github, before ADR 0029 repatriated the converter) and a truncated docstring sentence is finished. The ADR README now says what an 'Amendment YYYY-MM-DD:' block is for -- a clarification that leaves the verdict standing -- and that anything changing the verdict supersedes instead. * Restore the on008083 provenance in a comment on008083 is the real dataset behind #1197 -- 43 raw recordings, 2 stores, 36 failures, and five in no published list at all -- not a garbled token. Dropping it cost the comment its provenance and left a shape named after nothing. s3.ts and zarr-index-v3.test.ts both still cite it by id. * Type the arrow rows, catch the listing failure merge_index's failures param is list[FailureEntry]; converted, pending and prior_pending stay list[dict] with a note, since two of them are inputs this function does not validate and one is a worker-built store entry. PriorEventRows._row_group returned object and was then indexed and filtered, so both were unchecked accesses. Typed with an ArrowTable Protocol declaring the three members used: pyarrow is an optional lazy import, so there is no module-scope name to annotate with and a TYPE_CHECKING import would still depend on it being installed. purge --all no longer hands an operator a traceback when the listing fails. list_dataset_ids raising is right; letting it out of main was not. Same message shape and exit 2 as the empty case, with a test driving the stub aws failure injection. Also corrects a comment that described an 'or head' expression where the code is a ternary. * Correct two docs the earlier rounds left stale validated_workflows still called the engine-bump preview read-only with no writes; hallu-zarr.sh's own comment was corrected in round 1 and this was not. migrate_schema runs for every subcommand and its ALTER plus engine_version seed are idempotent, so a preview on a node that has never run the current schema writes those. Nothing is enqueued, parked or reset. catalog.ts said the contract declares zarr_data_failures CLOSED; it is .passthrough() since the B5 fix. The three-edit rule still holds -- a key this projection never reads is never in the object to begin with -- so the sentence now says that rather than resting on a strictness that is gone.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1206Version: 🤖 Updated on each push. |
This was referenced Sep 3, 2026
Closed
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reorganises the Zarr serving layer so a viewer, a library, or an agent can read a window in a bounded number of cacheable requests, with a documented index contract, honest coverage accounting, and a standing fidelity check. Producer-side changes ride one engine bump; the staging instance on Hallu proves the wave before production.
Closes #1181
Phases completed
?v=tokened metadata cached for a day.has_zarr, zarr fields on the wire,zarr_data_failuressummary declared,zarr-catalog.jsonpublished daily and on demand,GET /catalog.jsonon the contract host.hallu-zarr.sh --testwith six prod guard rails and--print-config; nightly cron on Hallu convertsnemar-db-dev's public datasets intonemar-dev, served byzarr-test.nemar.org.bids_channelson both entry points.discovered == stores + failures + pending,pending[]with carried attempts and a queue backoff,manifest.json, structurednemarprovenance attrs, JSON schemas served at/schemas/zarr-index-v3.json, top-level provenance andlayoutfor recipe-first clients, biosigio pinned>=1.2.7,ZARR_ENGINE_VERSION = "3".sweep_stamps, fail-open on infrastructure errors,has_zarr_verified,zarr_verify_statuson the contract and catalog.events.parquetbesideindex.json,sample_index = floor(onset * rate + 0.5)per store, rows for untouched stores carried from the prior file while a reconverted store never inherits them, validated against the schema before any upload, pointer and row count in the index and manifest, upload failures reported on the failed callback without failing the run;purge_non_raw_stores.pyre-reads the live index and writes it back with a conditionalIf-Match.Website side (nemarOrg/website#278, targets
staging): v1/v3 tolerant index parsing, a coverage panel with per-recording jump links, verification badge and units notice from the v3 provenance,has_zarrandhas_zarr_verifiedfilters; follow-ups nemarOrg/website#279 (Playwright harness) and #280 (zarr filters in search mode).Also filed and resolved along the way: #1197 (per-dataset coverage, landed in phases 2 and 7), #1191 (failures summary contract), neuromechanist/biosigio#122, #123, #127; later-effort #1198 (process report schema); nemarOrg/website#276 (closed by #278).
Ops after merge
hallu-zarr.sh --preview-engine-bump, thentouch /mnt/local/zarr-state/.zarr-engine-bump-ackfor exactly one run; watchengine_requeued.ZARR_DRIVER_REFfrom the epic branch todev; place.zarr-secrets.envin the test state dir./mnt/local/zarr-state/purge-nonraw-*.json. Re-check with the reproduce command on Zarr converter walks the whole repository, not the BIDS raw tree #1058 after the engine wave, since the wave re-reads every dataset, and re-measure Zarr fidelity: 12 datasets serve one-channel copies of multichannel EEG; verification gate + quarantine + re-conversion #1068.Test plan
xx099905through the test pipeline; the engine-3 re-conversion of the exemplar fleet is verified onzarr-test.nemar.orgbefore the production ack.Whole-epic review
Six Sonnet reviewers read the full dev...epic diff after every phase had been reviewed individually: backend code, converter and ops code, test coverage, silent failures, type design, and comments and ADRs. Every lens returned "mergeable to dev" with no critical finding; the findings below only show at the whole-epic level and are all fixed on #1208 (squash-merged into this branch), none deferred.
Must-fix, all fixed:
attempted_atinsidesweep_stamps(no new column, ADR 0034) and candidates are ordered never-attempted first, then oldest attempt. The JSON path is one exported constant shared by the sweep and its three readers, with a seam test through the catalog builder and thehas_zarr_verifiedfilter.pendinganddiscoveredintozarr_data_failures, but the detail route projected them away and the contract's nested schema stripped them on parse. Both now carrypending,discovered,events_upload_failed,manifest_upload_failed(the last two were previously visible only in aconsole.warn), with a callback-to-D1-to-detail-to-contract round-trip test.SkipTesthid 43 tests behind "skipped=9". The full converter suite now runs in the job that installs the real requirements, with a tripwire that fails above two expected skips..int();data_failureswas typed as a shaped array the validator never checked.round(); the code isfloor(x + 0.5), which differs at exact ties. Both now state the exact formula.Hardening from the should-fix tier, all fixed: index.json is now published with a conditional write (
If-Match, one re-read-and-retry on 412, then a loud failed callback; Hallu's aws-cli 2.35.20 supports it), a failed S3 listing raises instead of reading as "no datasets", ashared/**CI path filter gates lint, unit-pure and integration-dev, the engine-bump ack file lifecycle and the redirect-before-cache ordering each have a real-script or real-harness test, and the remaining items were comment and type-annotation fixes.Filed rather than fixed here: #1207 (validate live catalog and detail responses against the shared contract; the detail route serves a numeric
idwhere the contract says string), website#279 (Playwright harness), website#280 (Zarr filters in search mode).